1. Field of the Invention
The present invention relates to document creation and communication through the document. More specifically, the present invention relates to method and apparatus allowing addition and display of an electronic memo to a document no matter whether the document is electronic or paper document, as well as to a computer readable recording medium storing an electronic memo processing program for implementing the electronic memo processing method.
2. Description of the Background Art
Recently, it is a common practice to create a document by using an apparatus called a word processor, dedicated for document creation, or by using an application software for document creation executed on a computer. Such method is advantageous in that modification and addition to the document can readily be performed. In companies and offices, however, though a person in charge creates a document by using such an application program, the document thus created by the person in charge is rarely distributed intact. It is more general that the created document is once printed, proof-read by a supervisor or a colleague, the result of correction is reflected on the original document file and the document thus refined to higher completeness is then distributed. In such a case, the content of modification or comments are written by a pen on the printed copy of the document.
Such a method is advantageous in that portions and contents of modification can be recognized at one sight, and that proof reading can readily be done even by one not accustomed to computer operation, as the method does not require any computer operation. Further, document review on paper medium is advantageous in that a large number of pages can be read through in quite a short time period and that pages far apart from each other can readily be compared. Such advantageous are not available in a display device such as a computer.
The method, however, is disadvantageous in that it is difficult to integrally manage correspondence between the document stored in the computer and the modification memo written on the sheet of paper. It is often the case that a person other than the author does proof reading or creates comments on the document, and that several persons do such process on the identical document. In such a case, it is necessary to integrally manage the document stored in the computer and considerable amount of memos and comments written on the paper medium, for the document. In the conventional method, such an integrated management has been extremely difficult. In order to overcome such disadvantages, a system for integrally managing paper document, electronic document and a memo or comment created for the document has been desired.
Several proposals have been made to solve the above described problem, for a technique of adding a comment to a document which is being created or to refer to a comment added to the document, by using a common computer set. A typical example is a xe2x80x9csubstance/aliasxe2x80x9d model of TRON (The Real-Time Operating system Nucleus) project (first prior art). According to the technique, the data has data substance (substantive entity) and an icon (alias) and the alias is incorporated in an arbitrary data substance, whereby the data comes to have a function of a sort of hypertext.
Japanese Patent Laying-Open Nos. 1-026269, 1-035652 and 1-037656 disclose a method of document edition in which alternative character string for a character string in a document and information related thereto are stored, and the alternative character string is displayed as needed. A technique utilizing a xe2x80x9cdeskxe2x80x9d on which one works provided with a function of a computer, allowing communication between the one and the computer has been proposed in xe2x80x9cInteracting with Paper on the Digital Deskxe2x80x9d (Communications of the ACM. Vol. 36, No. 7, 1993) or in xe2x80x9cA Desk Which Responds to Operations on Real-World Objectsxe2x80x9d (Interactive System and Software, II-WISS""94, Kindaikagakusha, September, 1994) (Second Prior Art).
According to the second prior art, a camera is placed on a desk, movement of objects on the desk and movement of the user are monitored, and in accordance with the monitored situation, an image plane created by a computer is cast by a projector on the desk, thereby allowing communication between the computer and the user. When a user takes an action on an object on the desk or on the image plane cast by the projector, the function of the computer allotted to the object or the image plane is activated, and the result is projected on the desk. Especially in the article entitled xe2x80x9cInteracting with Paper on the Digital Deskxe2x80x9d, through the recognition of a graphic figure written on the desk, communication with the computer through paper medium is contemplated.
Japanese Patent Laying-Open No. 9-91301 proposes another technique attempting communication with the computer through paper medium (third prior art). This technique is to apply the concept of hypertext and hyper media, which has been a closed technique in the computer world, to paper document. For this purpose, according to the technique, document component to which related information has been added, is printed with some mark, for example, a screen. The user marks by a marker, a desired document component of the printed paper document. The paper document is scanned by a scanner to be input to the computer, and information related to the marked document component is output to the printer or to an output device such as a computer display.
When the document is to be output to a paper medium to be proof-read, it is necessary to print the document in plural copies, corresponding to the number of persons who will proof-read the document. This leads to increased printer paper consumption. When the content of the document is to be proof-read by a person at a remote distance, to expedite processing, it is necessary to transfer the document by a facsimile, for example, and to receive the proof-read copy again by utilizing a facsimile, for example, which procedure is inconvenient. Further, as already mentioned, it is difficult to integrally manage correspondence between the electronic document file stored in the computer and handwritten comments on the document printed on sheets of paper. It is convenient if works such as proof reading can be done without using paper medium, if possible. For that purpose, one cannot help but use a computer. However, it is desired for those not accustomed to computer operation in general and word processor software operation in particular, especially for those not good at input through keyboards, that the processing of the document can be done in the similar manner as on a paper medium.
Among the above described three prior art techniques, the first prior art technique may be considered a method of managing a memo added to the document. This technique is closed in a computer environment, and therefore, it is disadvantageous in that it cannot be applied to processing such as proof reading of the document on paper medium. In the modern society with much developed computer technique, almost everything which can be done using paper can also be done on the computer. Nevertheless, formidable amount of paper is still consumed and it is said that the paper consumption is ever increasing along with the wide spread use of the computers. The reason for this may be the better visual comprehension of the document written on a sheet of paper, especially the document printed out by the computer. This urges transfer of information from the computer to paper medium, to which information such as memo is added. Such new information added to the paper medium is not reflected on the original information in the computer, and rather, the information is diffused. Accordingly, there is an increasing demand of a technique which integrates paper medium and electronic medium which can be managed by a computer, and to integrally manage information such as memo added to the paper medium related to the electronic information.
In the second prior art technique, even when the memo written by the user on a sheet of paper is taken in the computer, correspondence between the memo and the paper medium on which the memo has been actually written is not stored. Therefore, integral management of the memo and the paper medium is not possible. Further, as the user directly writes the memo on the sheet of paper, if long memo is written, visual comprehension is degraded. Further, the device for this prior art technique is of large scale and not portable.
The third prior art technique contemplates communication with electronic medium through paper. However, the work of adding information to the document such as addition of a memo has to be done in the conventional closed world of the computer. Further, in order to draw information from the paper document, it is necessary for the user to directly mark the document on the paper. Further, the marked document must be read by a scanner. This may contaminate the document and if there are various and many items described on the document and marks are crowded, or if the marks are not clear enough, necessary information would not be read from the document, making it difficult to draw related information therefrom.
A function of adding data manually written by a pen input to a document file created by using an application program, as a mechanism allowing direct manual writing of electronic comment to an electronic document, has already been implemented by a conventional application program. In the application program, the user is allowed to add pen input data when operational mode is switched to xe2x80x9cdrawingxe2x80x9d mode, which is different from a document editing mode.
However, the function is closed in a single application and is not a general mechanism. Therefore, a document created by another application cannot be processed. This means that various and many applications must each have a mechanism for handling pen input data. Further, the memo created in this manner is integrated with the original document, and therefore the memo created in this manner cannot freely be taken out and used for other application programs. If a document file is created by one application and even if another application has a function of reading and displaying the document file, the layout of the document created by the aforementioned one application is not always reproduced as it is. Therefore, there would be documents of various different layouts with memos written therein. Therefore, it is difficult to integrally manage the memo and electronic documents.
Further, in the application of the prior art described above, the pen input data constitutes part of the document. Therefore, it is not possible to separately extract only the pen input data from other parts of the document and to use the pen input data for another purpose.
One solution of such problems has been proposed in Japanese Patent Laying-Open No. 5-189185. In the invention described in this laid-open application, an image data is drawn on a window. A transparent window is provided virtually on the drawn image data, and an image can be drawn without damaging the aforementioned image drawn on the window. Utilizing this mechanism, an image of a document file is first drawn, and a memo is drawn by a pen on the virtual transparent window, and in this manner, the user can operate with a feeling as if the user modifies, corrects or writes memo on a paper.
However, the method disclosed in Japanese Patent Laying-Open No. 5-189185 allows only drawing of another image without damaging the base image. This method is silent about a framework which allows a plurality of users at separate positions to write memos on one document file. Accordingly, the method as it is is not applicable to proof reading of a document, for example.
Therefore, an object of the present invention is to provide method and apparatus of electronic memo processing which allow integral management of a memo added to a document together with the document, regardless of the media, and to provide a computer readable recording medium storing an electronic memo processing program.
Another object of the present invention is to provide method and apparatus for an electronic memo processing which allow addition of an electronic memo to a document regardless of the medium, with a feeling as if a comment is written by a pen on a sheet of paper, thereby to facilitate participation of even a user not accustomed to computer operation, to document proof reading operation, and to provide a computer readable recording medium storing the electronic memo processing program.
A still further object of the present invention is to provide method and apparatus of electronic memo processing allowing addition of electronic memo to a paper document, and allowing integral management of the electronic memo with the original paper document, and to provide a computer readable recording medium storing the electronic memo processing program.
A still further object of the present invention is to provide method and apparatus of electronic memo processing allowing modification of a document file on an electronic medium with a feeling as if correction is made on a sheet of paper, and allowing document modification operation even by a user not accustomed to computer operation, and to provide a computer readable recording medium storing the electronic memo processing program.
A still further object of the present invention is to provide method and apparatus of electronic memo processing allowing addition of electronic comments by a plurality of users to a document regardless of the medium, and allowing easy distinction among these comments, and to provide a computer readable recording medium storing the electronic memo processing program.
The electronic memo processing apparatus in accordance with the present invention includes a document identifying device for identifying an arbitrary object document; an electronic memo retrieving device for retrieving, from a storing device, an electronic memo related to the document identified by the document identifying device; and a display device for displaying the memo on a virtual transparent input sheet registered or positioned to be overlapped with the object document.
As the electronic memo related to the object document is retrieved from a storage device and displayed registered to be overlapped with the object document, the electronic memo and document are viewed together as if the memo is written on the document.
Preferably, the electronic memo processing apparatus includes an electronic memo acquiring device for electronically acquiring electronic memo to a document, input by a user through a virtual transparent input sheet registered to be overlapped with the arbitrary object document, and an electronic memo storing device for storing the electronic memo acquired by the electronic memo acquiring device, associated with the object document, in a storage.
It is possible for the user to input electronic memo on a virtual transparent input sheet overlapped with the document, and to store the electronic memo by associating the memo with the document. Therefore, even a user not accustomed to operation of an input device such as a keyboard, can readily input the electronic memo to the document, and the electronic memo can be managed integrated with the document.
Preferably, the object document is a document on a paper medium, and the apparatus further includes an image pickup device for taking the object document as an image, an image recognizing device for recognizing the content of the document based on the image taken by the image pickup device, and a device for identifying the object document based on the content of the document recognized by the image recognizing device.
A camera, for example, may be used for taking in the document as an image. This allows identification of the document without the necessity of providing an identifier to the document.
Preferably, the object document is a document electronically created and thereafter printed on paper medium, and the electronic memo processing apparatus further includes a device for searching for and taking out an electronically created document corresponding to the printed document identified by the image recognizing device and the identifying device, from a document storage.
When an electronically created document is printed on a paper medium, it means that an electronic document corresponding to the document on the paper is stored in a document storage. Therefore, by taking out the electronically created document corresponding to the document identified by the image recognizing device and the identifying device from the document storage, it becomes possible to associate the electronic memo added to the document printed on the paper with the original electronic document. Therefore, integral management of the document and the electronic memo is allowed, utilizing paper medium also. Further, when a single electronic document is printed in plural copies, proof-read and modified by a plurality of persons through electronic memos, it is possible to integrally manage the plurality of electronic memos and the original electronic document.
Preferably, the electronic memo processing apparatus further includes a modification detecting device for detecting how the object document is modified after last addition of an electronic memo, and a layout adjusting device for adjusting position of display of the electronic memo on the object document, in accordance with the layout modification of the object document detected by the modification detecting device.
If the layout of the document is modified and the position where the memo is to be displayed is kept as it is, the memo would be displayed at a portion different from that portion where the memo has been initially added on the document. Therefore, if the layout of the document is modified, the position of display of the memo is changed in accordance with the layout modification, so that the memo is always displayed at a light position.
More preferably, the object document is a document on an electronic medium and, more preferably, the electronic memo processing apparatus in accordance with the present invention further includes a device for retrieving an object document from the document storage and for creating a document image; a device for creating an image of an electronic memo retrieved by the electronic memo retrieving device; and a selective synthesizing device for displaying arbitrary one of the document image and the image of the electronic memo, or displaying the document and memo images synthesized, in accordance with a user""s designation.
If the object document is an electronic document, it is possible to create a document image by retrieving the document from the document storage using a document retrieving device, without the necessity of printing the document. Thus formed document image may be displayed synthesized with the electronic memo, or either one of the images of the document and the memo may be displayed by itself. Therefore, the document and the electronic memo can be displayed in any form among the form of the original document alone, the form of the electronic memo alone, and the form when the document is printed on a paper medium and the memo is written thereon. Therefore, as compared with the memo simply written on the paper medium, it is possible to browse and manage the document and the memo more effectively.
Preferable, the format of the electronic memo stored in the electronic memo storing device includes a plurality of electronic memos associated with the object document, and each of the plurality of electronic memos includes attribute data. The display device displays the plurality of electronic memos in different manners of display dependent on the attributes. In this case, xe2x80x9cmanner of displayxe2x80x9d includes xe2x80x9cnot displayingxe2x80x9d, dependent on the attribute.
As the electronic memo is displayed in different manner of display in accordance with the attribute, the electronic memos can be displayed and readily distinguished from each other on the basis of who created the memos, where the memos are created, when the memos are created or by what electronic memo processing apparatus the memos are created.
Preferably, the object document is an electronic document, and the electronic memo processing apparatus further includes an operator recognizing device for electronically acquiring and recognizing an operator for the document, which is manually input by a user through the virtual transparent input sheet; and a document modification device for modifying the electronic document in accordance with the operator acquired by the operator recognizing device.
When the user manually inputs an operator for the document on the virtual transparent input sheet displayed overlapped on the document, the operator is recognized, and the electronic document is modified in accordance with the operator. The electronic document can be modified easily in an intuitive, readily comprehensible manner, as compared with the use of a keyboard, for example.
More preferably, the electronic memo processing apparatus further includes a modification detecting device for detecting how a layout of the object document is modified by the document modifying device, and a layout adjusting device for adjusting position of display of the electronic memo of the object document, in accordance with the layout modification of the object document detected by the modification detecting device.
If the layout of an electronic document is modified and the position of display of the memo is kept as it is, the memo would be displayed at a position different from that position at which the memo is originally added to the document. In order to avoid such inconvenience, if the document layout is modified, the position of display of the memo is changed in accordance with the layout modification, so that the memo is always displayed at a right position.
According to another aspect of the present invention, the electronic memo processing apparatus includes a display device for displaying an electronic document; an operator recognizing device for electronically acquiring and recognizing an operator for the document, manually input by a user on a virtual transparent input sheet registered to be overlapped with the object document; and a device for modifying the electronic document in accordance with the operator acquired by the operator recognizing device.
When the user manually inputs an operator for the document on a virtual transparent input sheet displayed overlapped with the electronic document, the operator is recognized and the electronic document is modified in accordance with the operator. Therefore, the electronic document can be modified easily in more intuitive, readily comprehensible manner, as compared with the use of a keyboard, for example.
According to a still further aspect of the present invention, the electronic memo processing method includes the steps of identifying an object document; retrieving an electronic memo associated with the document identified in the step of document identification from a storage; and displaying the electronic memo searched in the step of searching on a virtual transparent input sheet registered to be overlapped with the object document.
Preferably, the electronic memo processing method further includes the steps of: electronically acquiring an electronic memo to a document input by a user through a virtual transparent input sheet registered to be overlapped with an arbitrary object document; and storing the electronic memo acquired in the step of acquiring the electronic memo in storage, by associating the memo with the object document.
Preferably, the step of display includes the step of displaying the electronic memo on a binocular type display device having transparent display portion, displaying the object document and track of manually written data overlapped with each other, when worn by the user.
More preferably, the object document is a document on a paper medium, and the step of identifying the document includes the steps of: taking the object document as an image; recognizing content of the document based on the image taken in the step of taking the document as an image; and identifying the object document based on the content of the document recognized by the step of recognizing the image.
Preferably, the object document is a document created electronically and thereafter printed on a paper medium, and the electronic memo processing method further includes the step of searching for and taking an electronically created document corresponding to the document identified by the step of recognizing the image, from document storage.
More preferably, the electronic memo processing method further includes the steps of: detecting how layout of the object document is modified after last addition of an electronic memo to the object document; and adjusting position of display of the electronic memo for the object document, in accordance with the layout modification of the object document detected in the step of detecting modification.
Preferably, the object document is a document on an electronic medium, and the electronic memo processing method further includes the steps of creating a document image by retrieving the object document from the document storage, and displaying the document image synthesized with the searched electronic memo in the step of displaying the electronic memo. In the step of synthesizing and displaying, the document image and the image of the electronic memo may be synthesized by obtaining exclusive OR of the document image and the image of the electronic memo retrieved in the step of displaying the electronic memo.
More preferably, the object document is an electronic document and the electronic memo processing method further includes the steps of: electronically acquiring and recognizing an operator for the document manually input by the user through a virtual transparent input sheet; and modifying the electronic document in accordance with the operator acquired in the step of recognizing the operator.
Preferably, the electronic memo processing method further includes the steps of: detecting how a layout of the object document is modified in the step of document modification; and adjusting position of display of the electronic memo for the object document, in accordance with the layout modification of the object document detected in the step of detecting modification.
According to a still further aspect of the present invention, the electronic memo processing method includes the steps of: displaying an electronic document; electronically acquiring and recognizing an operator for the document manually input by a user through a virtual transparent input sheet registered to be overlapped with the object document; and modifying the electronic document in accordance with the operator acquired in the step of recognizing the operator.
According to a still further aspect of the present invention, the recording medium is a computer readable recording medium storing a computer program for implementing the electronic memo processing method described above.
The foregoing and other objects, features, aspects and advantages of the present invention will become more apparent from the following detailed description of the present invention when taken in conjunction with the accompanying drawings.